Repository navigation
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
1 Skipped Deployment
|
| const researchSessionStateSchema = z.object({ | ||
| researchRetrievedUrls: z.array(z.string()).default([]), | ||
| researchToolAttempts: z.record(z.number()).default({}), | ||
| }); |
There was a problem hiding this comment.
⚠️ Bug: Retrieved-URL and attempt state is session-scoped and never reset per run
researchRetrievedUrls and researchToolAttempts live in ctx.session.state and nothing clears them at the start of a run action (no other reference exists in the repo). In a session with a second question, the validator treats every URL retrieved in earlier questions as valid provenance, so a synthesizer can cite a URL from a previous run and still pass validateResearchReport. The returned retrieval.urls and retrieval.attempts also accumulate across runs, so they no longer describe the current report. Reset both fields at the start of the run, e.g. a leading .tap handler in research-run that patches { researchRetrievedUrls: [], researchToolAttempts: {} }. Alternatively, keep them in sequencer or run-scoped state.
Clear provenance state at the start of each research run.:
const resetProvenance = handler({
name: "research-reset-provenance", inputSchema: z.unknown(), sessionStateSchema: researchSessionStateSchema,
execute: async (_i, ctx) => { await ctx.session.patchState({ researchRetrievedUrls: [], researchToolAttempts: {} }); },
});
const run = sequencer({ name: "research-run", inputSchema })
.tap(resetProvenance).map((input) => ({ goal: input.question })).step(researchBlock);
- Apply fix
Check the box to apply the fix or reply for a change | Was this helpful? React with 👍 / 👎
There was a problem hiding this comment.
Addressed in 0ad3919: research provenance (retrieved URLs and attempts) now resets at the start of each run. CI green on that commit.
— FSD Head of Engineering
| maxTasks: z.number().int().positive().default(12), | ||
| maxAttemptsPerTask: z.number().int().min(2).default(2), | ||
| maxIterations: z.number().int().positive().default(3), | ||
| maxTotalRetries: z.number().int().nonnegative().default(6), |
There was a problem hiding this comment.
💡 Bug: Retry control on maxAttemptsPerTask and schema/docs mismatch
maxAttemptsPerTask is forced to min(2), so callers cannot disable per-task retry, which contradicts the 'retryable' framing for apps that want a single attempt. maxTotalRetries defaults to 6 but can be set below what maxAttemptsPerTask implies, and that interaction is not validated or documented. Either allow min(1) or document the constraint on the config fields, and validate the relationship between the two settings.
Was this helpful? React with 👍 / 👎
There was a problem hiding this comment.
Addressed in 0ad3919: maxAttemptsPerTask now allows a single attempt, and the changeset/docs were aligned. CI green on that commit.
— FSD Head of Engineering
| import { describe, expect, it } from "vitest"; | ||
| import { validateResearchReport } from "../src/research"; | ||
|
|
||
| describe("research report citations", () => { | ||
| it("rejects citations that no completed research task retrieved", () => { | ||
| expect(() => | ||
| validateResearchReport( | ||
| { | ||
| question: "What changed?", | ||
| summary: "A sourced answer.", | ||
| findings: [{ claim: "The change shipped.", citations: ["https://invented.example/report"] }], | ||
| sources: [{ title: "Invented", url: "https://invented.example/report" }], | ||
| }, | ||
| new Set(["https://fixture.example/source"]), | ||
| ), |
There was a problem hiding this comment.
💡 Quality: Missing tests for kit install, config validation and tool provenance
The unit tests only cover validateResearchReport. Nothing tests the config .refine (model or role-specific models required), retrievedUrls extraction for search, fetch and crawl, observableTool attempt counting and recording, or the executor-source provenance check in research-validate-report. Those are the core of the kit's provenance claim and rely solely on the goal script. Add testBlock unit tests per the repo's testing guidance, and run the goal with fsdev run as the verification path.
Was this helpful? React with 👍 / 👎
There was a problem hiding this comment.
Addressed in 0ad3919: added tests in packages/patterns/test/research.test.ts (+160 lines) beyond validateResearchReport. CI green on that commit.
— FSD Head of Engineering
|
Note Automatic reviews are paused because your team has used its included automatic processing for this billing period (headroom scales with your seat count). You can still comment "Gitar review" to run one anytime, and automatic reviews resume on their own by December 1. Add seats for more headroom. Code Review
|
Mechanism check (advisory, never blocks)Rows from
If an export is a new mechanism, add a row. CI filled the mechanical fields; - id: TODO
job: TODO
package: "@flow-state-dev/patterns"
anchor: validateResearchReport
file: packages/patterns/src/research/index.ts
builtOn: [] # TODO
overlaps: [] # TODO
status: canonical
doc: TODO
- id: TODO
job: TODO
package: "@flow-state-dev/patterns"
anchor: research
file: packages/patterns/src/research/index.ts
builtOn: [] # TODO
overlaps: [] # TODO
status: canonical
doc: TODO |
…dd tests (FIX-2822)
Closes FIX-2822
Issue: #5422 (epic FIX-2813 Showcase kits)
Codex task: https://chatgpt.com/codex/tasks/task_e_6afd14d891388324a58b92efaf1fcff1
What was built
Adds
research()inpackages/patternsbuilt on plan-and-execute with search, fetch and crawl, per-task retry/rescue, and citation validation against retrieved sources. Adds tests, goalresearch-returns-a-cited-report, changeset, lockfile/tsconfig updates.How the goal proves it
See the goal under
goals/kits/added in this PR (run line in its goal.md). It runs the installed app (not the kit factory) with deterministic/scripted models and includes GOAL_CONTROL negative controls that must fail.Generated by Codex Cloud; applied from the task diff unchanged. Not yet run locally or in CI-verified — draft.